This adds a new repo index subcommand to spk for index generation and updates. It adds the --use-indexes and --no-indexes flags for repository index usage. This also updates resolvo solver to get global variables data from an indexed repository. This allows resolvo to solve without needing to restart its solves.

Indexing

The index is designed to help the solvers with solve times. It doesn't contain enough data to help with other spk operations like building and testing a package.

Indexing can be enabled or disabled in the spk config file. If indexing is enabled, you have to generate an index, with spk repo index, prior to trying to use it. They are not generated on the fly (outside of automated tests).

To generate an index (for the origin):

• spk repo index --disable-repo local

To update an existing index, e.g. after a new python package was published:

• spk repo index --disable-repo local --update python

The flatbuffer index data is stored in a file in the underlying spfs repo in a index/spk/ sub-directory.

If index use is enabled in the config file, it can be disabled with the --no-indexes command line flag. If index use is disabled by default, it can be enbled with the --use-indexes flag. If index use is enabled, but no index has been generated, spk will fallback to using the underlying repo directly (it acts as it would before this change).

Speed Diferences

Generating the index file on our repo (sizes below) takes about 2 minutes. Updating a package in an existing index, such as after a new build is published, takes a few seconds.

Sample solver time improvements using this indexing

The numbers come from this setup:

• an origin repo that has 2245 packages, 23540 versions, 82517 builds (11 erroring, about 30% deprecated), and 141 global vars
• the index loads in ~0.0003 seconds unverified, or ~0.2 seconds verified, and is about 76 MB on disk with trimmed down deprecated builds (107 MB with full deprecated builds)
• these times are from a rough average of 3-4 runs with index verification disabled
• a "toolset" below is a set of requests for the named DCC and our typical in-house plugins and tools

Requests        | Solution size | Num.    | Solve time  |  Indexed solve time, no retries
                | (# packages)  | Retries | (seconds)   |  (seconds)
-----------------------------------------------------------------------------------------
python          |        4      |    1    |     0.17    |   0.03 
boost-python    |        8      |    1    |     0.31    |   0.05 
python-torch    |       37      |    2    |     0.58    |   0.15 
widget toolset  |       60      |    2    |     3.44    |   0.48 
katana toolset  |      181      |   10    |    18.32    |   2.97 
nuke toolset    |      280      |   12    |    24.32    |   4.55 (*)
houdini toolset |      211      |   19    |    37.24    |   6.58 (*)
maya toolset    |      403      |   20    |    59.50    |   9.52 (*)

The indexing doesn't have a noticable impact (to users) on smaller solves. But it allows our larger solves to finish in under 10 seconds, or about 1/6th of the time they currently do. The times marked with (*) are improved further by the changes in PR6: (#1344).

This is the final 5 of 5 chained PRs for adding indexes to spk solves:

1. Indexes 1: Change Package and related traits to not return references to fields #1336
2. Indexes 2: Add new_unchecked() constructors to spk schema objects #1337
3. Indexes 3: Adds flatbuffers schema and SolverPackageSpec for indexes to spk #1338
4. Indexes 4: Adds Indexes for SPK repositories #1339
5. this PR
6. Indexes 6: Changes version_filter field in index schema #1344
7. Indexes 7: Adds a lock file around index generation and updates #1354
8. Indexes 8: Fixes the spk build or mkb crash with indexes enabled #1355
9. Indexes 9 - Adds messaging on package events to kafka #1356
10. Indexes 10: Adds index update listeners and Indexer #1363